Modelling Paralinguistic Conversational Interaction Towards social awareness in spoken human-machine dialogue

Parallel with the orthographic streams of words in conversation are multiple layered epiphenomena, short in duration and with a communicative purpose. These paralinguistic events regulate the interaction flow via gaze, gestures and intonation. This thesis focus on how to compute, model, discover and analyze prosody and it’s applications for spoken dialog systems. Specifically it addresses automatic classification and analysis of conversational cues related to turn-taking, brief feedback, affective expressions, their crossrelationships as well as their cognitive and neurological basis. Techniques are proposed for instantaneous and suprasegmental parameterization of scalar and vector valued representations of fundamental frequency, but also intensity and voice quality. Examples are given for how to engineer supervisedlearned automata’s for off-line processing of conversational corpora as well as for incremental on-line processing with low-latency constraints suitable as detector modules in a responsive social interface. Specific attention is given to the communicative functions of vocal feedback like "mhm", "okay" and "yeah, that’s right" as postulated by the theories of grounding, emotion and a survey on laymen opinions. The potential functions and their prosodic cues are investigated via automatic decoding, data-mining, exploratory visualization and descriptive measurements.